Papers with LM-internal representations
Are representations built from the ground up? An empirical examination of local composition in language models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Compositionality is a hallmark of human language, but many phrases are non-compositional . a study by a team of researchers shows that LMs may not be able to distinguish between compositional and non-composable phrases. |
| Approach: | They propose to predict LM-internal representations of longer phrases given their constituents . they find that the representation of a parent phrase can be predicted with some accuracy . |
| Outcome: | The proposed model can predict a parent phrase with some accuracy given its children's transformations, but this is not the case. |